$d
[1] -1.077545
$dc
[1] 0.7020467
$bc
[1] 3.273021
$se
[1] 1.601865
$df
[1] 29.65953
$g
[1] -1.050026
$gc
[1] 0.6841176
The critical effect size value is the smallest effect that can reach statistical significance given the test, the sample size, and \(\alpha\).
Example: For a two-sided test of a correlation with n = 20, the critical r = 0.44. Effects between −0.44 and +0.44 won’t be significant.
The bigger the sample, the smaller the critical effect size.
Before conducting the study:
In front of the dead corpse:
And more..
We developed an r package that implements the computation of critical effect size values for correlation, groups comparison, linear regression and meta-analysis.
$d
[1] -1.077545
$dc
[1] 0.7020467
$bc
[1] 3.273021
$se
[1] 1.601865
$df
[1] 29.65953
$g
[1] -1.050026
$gc
[1] 0.6841176
Welch Two Sample t-test
data: Holz$t01_visperc[Holz$female == 1] and Holz$t01_visperc[Holz$female == 2]
t = 1.4095, df = 298.9, p-value = 0.1597
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.06419229 0.38823118
sample estimates:
mean of x mean of y
4.314090 4.152071
|== Effect Size and Critical Value ==|
d = 0.1623665 dc = ± 0.2269609 bc = ± 0.2262117
g = 0.1619587 gc = ± 0.2263909
Aims:
Demonstrate the practical utility of critical effect size values.
Investigate the cumulative progress of a research field.
How:
By computing critical effect size values for individual studies within selected meta-analyses, to test whether those studies could have detected a significant effect equal to the meta-analytic estimate.
Assessing the cumulative nature of research my investigating if future studies plan sample sizes based on previous meta-analytic effects.
Focus on main effects
Available table of coded studies
Published after the replicability crisis became a thing
Not too recent as well
one with large sample size for individual studies
one with small sample size for individual studies
The meta-analysis investigates the association between social media use and depressive symptoms in young adolescents.
We have calculated critical correlations for the individual studies and compared them to the meta-analytic effect.
In this case, all individual studies had critical effect size values smaller than the meta-analytic effect (100%).
If the meta-analytic effect truly exists, each study can reach significance for it.
In areas with large and easy to collect samples, null results are less likely to stem from insufficient power.
This meta-analysis investigates differences between musicians and non-musicians in working memory tasks expressing the effect is Hedge’s g due to the small samples.
It has 3 meta-analyses:
long term memory,
short term memory
working memory
We will compute critical values for two tailed t-test, assuming alpha at the .05 threshold and plot them comparing them to the meta-analytic effect.
In 0% of cases, the critical value was lower than the meta-analytic effect size, indicating that none of the studies in the meta-analysis could have detected significance for the meta-analytic effect.
When comparing the critical values to the bias corrected meta-analytic effect size the percentage remains 0%, as it cannot go any lower.
In 30% of cases, the critical value was lower than the meta-analytic effect size, indicating that 70% of the studies couldn’t reach significance for the meta-analytic effect.
When comparing the critical values to the bias corrected meta-analytic effect size the percentage lowered to 0%.
In 15.79% of cases, the critical value was lower than the meta-analytic effect size, indicating that these studies would have been able to detect a statistically significant effect of that magnitude.
In this meta-analysis no bias was detected.
Of the 214 works citing the meta-analysis, 121 had an accessible manuscript.
Of those 121 manuscripts, 21 had a research design that was compatible with the meta-analysis.
Of those 21, only 2 had run a power analysis but they did not choose the effect size based on that meta-analysis.
A total of 22 manuscripts out of 121 reported a power analysis.
In other 3 meta-analysis (Adesope et al., 2010; Harkin et al., 2016; Miller et al., 2015) we found that only 34.38% of individual studies had the chance to find a significant result assuming the meta-analytic effect to be true.
One of the studies had a meta-analytic effect of 1.16, if this one is removed the percentage drops to 13.12%.
Selecting arbitrarily 3-4 meta-analyses is not enough to drag conclusions on a field
We will systematically select at least 15 meta-analysis from 2016
We will look into papers citing those meta-analysis as we did for Talamini et al. (2017)
How many meta-analysis would be enough to drag conclusions on the matter?
What is the percentage of problematic studies that would convince you that the problem is real?
If you are interested in our first paper on Critical Effect Size Values:
Stay tuned for the next one ;)
ambra.perugini@phd.unipd.it